Papers with training speed
Punctuation-Steered Representation Fine-Tuning (2026.acl-short)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning (PeFT) are limited due to their prohibitive size and computational demands. |
| Approach: | They propose a method that fine-tunes punctuation representations to achieve performance improvements. |
| Outcome: | The proposed method improves performance by altering the representation space alone . but it results in suboptimal performance due to the effects of the method on the output . |
BCTH: A Novel Text Hashing Approach via Bayesian Clustering (2020.aacl-main)
Copied to clipboard
| Challenge: | Similarity search is a promising strategy to find the most similar items for a given query item. |
| Approach: | They propose to utilize Bayesian Clustering for Text Hashing to map documents to binary codes by utilizing multiple Bayessian Clusters in parallel. |
| Outcome: | The proposed approach is competitive compared with baselines in the perspective of precision and training speed. |
Making Asynchronous Stochastic Gradient Descent Work for Transformers (D19-56)
Copied to clipboard
| Challenge: | Asynchronous stochastic gradient descent (SGD) converges poorly for Transformer models . synchronous SGD is faster at raw training speed since it avoids waiting for synchronization . |
| Approach: | They propose a method to restore convergence by summing several asynchronous updates instead of applying them immediately. |
| Outcome: | The proposed method achieves the same BLEU score 1.36 times faster than asynchronous SGD. |
Joint Semantic and Distributional Word Representations with Multi-Graph Embeddings (D19-53)
Copied to clipboard
| Challenge: | Prior work has shown that word embeddings can be improved by using semantic knowledge-bases. |
| Approach: | They propose a way to combine distributional and semantic information while preserving lexical information of co-occurrences of words. |
| Outcome: | The proposed method improves word embeddings on a variety of word similarities. |
FastFit: Fast and Effective Few-Shot Text Classification with a Multitude of Classes (2024.naacl-demo)
Copied to clipboard
| Challenge: | Few-shot prompting of large language models (LLMs) via API calls presents a unique challenge when dealing with a multitude of classes that share similar semantic meanings. |
| Approach: | They present a Python package that integrates batch contrastive learning and token-level similarity score to provide fast few-shot classification. |
| Outcome: | The proposed method significantly improves multi-class classification speed and accuracy across English and Multilingual datasets. |
CytonMT: an Efficient Neural Machine Translation Open-source Toolkit Implemented in C++ (D18-2)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) has made remarkable progress over the past few years. |
| Approach: | They propose to use C++ and NVIDIA’s GPU-accelerated libraries to build an open-source neural machine translation toolkit called CytonMT. |
| Outcome: | The proposed toolkit accelerates the training speed by 64.5% to 110.8% on neural networks of various sizes, and achieves competitive translation quality. |
Interactive Text Games: Lookahead Is All You Need! (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing approaches to ground LLMs in textual interactions have been limited due to low computational efficiency and limited performance. |
| Approach: | They propose to use Lookahead models to ground LLMs in interactive text-based games to investigate their language grounding capabilities. |
| Outcome: | The proposed model significantly improves training speed and performance relative to the size of the action space. |
Dense Template Retrieval for Customer Support (2022.coling-1)
Copied to clipboard
| Challenge: | Templated answers are used to cover a wide range of topics, but the number of templates is often too high for an agent to manually search. |
| Approach: | They propose a dense retrieval framework that adapts a standard in-batch negatives technique to support unpaired sampling of queries and templates. |
| Outcome: | The proposed approach improves performance and training speed over more standard methods. |
Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt tuning is parameter-efficient but lags behind other state-of-the-art methods. |
| Approach: | They propose a parameter-efficient tuning method that only optimizes a soft prompt to adapt PTMs to downstream tasks. |
| Outcome: | The proposed method is parameter-efficient but lags behind other state-of-the-art methods. |
Cross-lingual Transfer Learning with Data Selection for Large-Scale Spoken Language Understanding (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to improve cross-lingual transfer learning on spoken language are pre-train on all available supervised data from another language. |
| Approach: | They propose a language model based source-language data selection method for cross-lingual transfer learning in spoken language understanding. |
| Outcome: | The proposed method reduces training time and improves model performance on spoken language understanding. |
ECHO-LLaMA: Efficient Caching for High-Performance LLaMA Training (2025.emnlp-industry)
Copied to clipboard
Maryam Dialameh, Rezaul Karim, Hossein Rajabzadeh, Omar Mohamed Awad, Boxing Chen, Hyock Ju Kwon, Walid Ahmed, Yang Liu
| Challenge: | ECHO-LLaMA transforms LLa MA models into shared KV caching across certain layers, significantly reducing KV computational complexity while maintaining or improving language performance. |
| Approach: | They propose an efficient LLaMA architecture that transforms LLama models into shared KV caching across certain layers, reducing computational complexity while maintaining or improving language performance. |
| Outcome: | ECHO-LLaMA achieves up to 77% higher token-per-second throughput during training, up to 16% higher Model FLOPs Utilization (MFU) and up to 14% lower loss when trained on an equal number of tokens. |
Span-based Hierarchical Semantic Parsing for Task-Oriented Dialog (D19-1)
Copied to clipboard
| Challenge: | Existing semantic parsers score intents and slots as labels of nesting nodes, but decode a valid tree globally. |
| Approach: | They propose a span-based semantic parser for parsing compositional utterances into Task Oriented Parse (TOP) the parsers score labels of the tree nodes covering each token span independently, but decode a valid tree globally. |
| Outcome: | The proposed parser outperforms previous methods on the TOP dataset in accuracy and training speed. |
Training Text-to-Text Transformers with Privacy Guarantees (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in NLP often stem from large transformer-based pre-trained models. |
| Approach: | They propose differentially private (DP) training as a potential mitigation for models that can memorize parts of training data. |
| Outcome: | The proposed model can memorize parts of training data and mitigate memorization concerns. |
Continual and Multi-Task Architecture Search (P19-1)
Copied to clipboard
| Challenge: | Recent studies have shown that architecture search can improve performance on language modeling and image classification tasks with reasonable training speed. |
| Approach: | They propose a continual architecture search approach that continually evolves the model parameters during sequential training of several tasks without losing performance on previously learned tasks. |
| Outcome: | The proposed approach improves language modeling and image classification with reasonable training speed and a weight-sharing strategy. |
FENAS: Flexible and Expressive Neural Architecture Search (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent approaches to architecture search have shown good improvements in terms of performance with reasonable training speed. |
| Approach: | They propose an algorithm with more activation functions, input edges, and atomic operations to search for architectures that are optimal for given task. |
| Outcome: | The proposed algorithm reproduces well-known LSTM and GRU architectures and initializes with them for finding architectures more efficiently. |
MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can be fine-tuned to new tasks, but in multi-task scenarios, training imbalance and seesaw effect often arise. |
| Approach: | They propose a flexible fine-tuning framework that leverages asymmetric optimization among LoRA experts to reduce training imbalance and improve performance. |
| Outcome: | The proposed framework outperforms baseline methods in inter- and intra-task learning scenarios. |
FastClass: A Time-Efficient Approach to Weakly-Supervised Text Classification (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent research shows keyword-driven methods can achieve state-of-the-art performance on various tasks. |
| Approach: | They propose an efficient weakly-supervised text classification approach using unlabeled data . they use dense text representation to retrieve class-relevant documents from unlabed corpus . |
| Outcome: | The proposed weakly-supervised classification method outperforms keyword-driven models on a wide range of classification tasks. |
Neural Language Modeling for Named Entity Recognition (2020.coling-main)
Copied to clipboard
| Challenge: | Experimental results show that named entity recognition systems are faster and more flexible for the size of the corpus. |
| Approach: | They propose to use a neural language model as an alternative to the conditional random field layer for named entity recognition. |
| Outcome: | The proposed system has a significant speed advantage with a marginal performance degradation. |
Efficient Training of Language Models with Compact and Consistent Next Token Distributions (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train language models have focused on maximizing the likelihood of the next token . however, the construction and querying of such n-grams can be costly and impede training speed. |
| Approach: | They propose a method to train language models faster by pre-aggregating corpus with collapsed n-gram distribution. |
| Outcome: | The proposed model improves model quality and convergence rate while reducing variance across mini-batches compared to the standard next-token loss method. |
Safe-FedLLM: Delving into the Safety of Federated Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing work on federated learning for large language models (FL) addresses privacy and data-silo issues in the training of large language model training. |
| Approach: | They propose a probe-based defense framework for FedLLM that constructs defenses across three levels: Step-Level, Client-Level and Shadow-Level. |
| Outcome: | The proposed framework improves FedLLM's robustness against malicious clients while maintaining competitive performance on benign data. |
On-policy Reinforcement Fine-tuning with Offline reward for Multi-step Embodied Planning (2026.acl-long)
Copied to clipboard
| Challenge: | Embodied planning requires agents to make coherent multi-step decisions based on dynamic visual observations and verbal goals. |
| Approach: | They propose an On-policy Reinforcement fine-tuning framework with offline rewards for Embodied Task Planning that preserves generalization benefits of RFT while addressing costly interaction and sparse rewards. |
| Outcome: | The proposed framework outperforms closed-source and online-RL methods on EmbodiedBench, a recent benchmark for interactive embodied tasks. |
Reinforcement Learning for Diffusion LLMs via Energy-Based Gibbs Alignment (2026.acl-long)
Copied to clipboard
| Challenge: | Diffusion Large Language Models (dLLMs) offer parallel decoding and bidirectional context modeling . aligning dLLms with reinforcement learning (RL) remains a challenge . |
| Approach: | They propose a variational framework that reformulates RL for dLLMs as a distribution matching problem. |
| Outcome: | The proposed framework reformulates RL for dLLMs as a distribution matching problem. |